Integration and classification of spatial data for 3D modelling and monitoring of built heritage

L'abstract è presente nell'allegato / the abstract is in the attachmen

Segmentación semántica multiclase en la digitalización del patrimonio mueble utilizando técnicas de aprendizaje profundo

[EN] Digitisation processes of movable heritage are becoming increasingly popular to document the artworks stored in our museums. An increasing number of strategies for the three-dimensional (3D) acquisition and modelling of these invaluable assets have been developed in the last few years, to efficiently respond to this documentation need and contribute to deepening the knowledge of the masterpieces investigated constantly by researchers operating in many fieldworks. Nowadays, one of the most effective solutions is represented by the development of image-based techniques, usually connected to a Structure-from-Motion (SfM) photogrammetric approach. However, while the acquisition of the images is relatively rapid, it is the processes connected to the data processing that are very time-consuming and require substantial manual involvement of the operator. The development of deep learning-based strategies can be an effective solution to enhance the level of automatism. In the case of the current research, which has been carried out in the framework of the digitisation of a collection of wooden maquettes stored in the ‘Museo Egizio di Torino’ using a photogrammetric approach, an automatic masking strategy using deep learning techniques is proposed, to increase the level of automatism and therefore, optimise the photogrammetric pipeline. Starting from a manually annotated dataset a neural network has been trained to automatically perform a semantic classification with the aim to isolate the maquettes from the background. The proposed methodology has allowed obtaining automatically segmented masks with a high degree of accuracy. The followed workflow is described (as regards acquisition strategies, dataset processing, and neural network training), and the accuracy of the results is evaluated and discussed. In addition, the possibility of performing a multiclass segmentation on the digital images to recognise different categories of objects in the images and define a semantic hierarchy is proposed to perform automatic classification of different elements in the acquired images.[ES] Los procesos de digitalización del patrimonio mueble son cada vez más populares para documentar las obras de arte almacenadas en nuestros museos. En los últimos años se han desarrollado un número creciente de estrategias de adquisición y modelado tridimensional (3D) de estos activos de valor incalculable, que responden de manera eficiente a esta necesidad de documentación y contribuyen a profundizar en el conocimiento de las obras maestras investigadas constantemente por investigadores que operan en muchos trabajos de campo. Hoy en día, una de las soluciones más efectivas está relacionada con el desarrollo de técnicas basadas en imágenes, generalmente conectadas a un enfoque fotogramétrico de estructura-y-movimiento (SfM). Sin embargo, si bien la adquisición de las imágenes es relativamente rápida, son los procesos relacionados con el procesamiento de los datos los que consumen mucho tiempo y requieren una participación manual sustancial del operador. El desarrollo de estrategias basadas en el aprendizaje profundo puede ser una solución eficaz para mejorar el nivel de automatismo. En el caso de la presente investigación, que se ha llevado a cabo en el marco de la digitalización de una colección de maquetas de madera almacenadas en el 'Museo Egizio di Torino' mediante un enfoque fotogramétrico, se propone una estrategia de enmascaramiento automático mediante técnicas de aprendizaje profundo, que incrementa el nivel de automatismo y por tanto optimiza el flujo fotogramétrico. A partir de un conjunto de datos anotados manualmente, se ha entrenado una red neuronal que realiza automáticamente una clasificación semántica con el objetivo de aislar las maquetas del fondo. La metodología propuesta ha permitido obtener más caras segmentadas automáticamente con alto grado de precisión. Se describe el flujo de trabajo seguido (en cuanto a estrategias de toma, procesamiento del conjuntos de datos y entrenamiento de las redes neuronales), y se evalúa y discute la precisión de los resultados. Además, se propone la posibilidad de realizar una segmentación multiclase sobre las imágenes digitales que permitan reconocer diferentes categorías de objetos en las imágenes y definir una jerarquía semántica que clasifique automáticamente diferentes elementos en la toma de las imágenes.The authors thank Volta® A.I. (and in particular Silvio Revelli) for the contribution to this work and for providing high-end hardware for neural network training. In addition, they would like to thank Alessia Fassone of Museo Egizio di Torino and all the people involved in the B.A.C.K. TO T.H.E. F.U.T.U.RE. project (in particular, Fulvio Rinaudo, who coordinated the Geomatic team). Finally, they wish to express their gratitude to Nannina Spanò and Filiberto Chiabrando for the helpful confrontation during the presented research.Patrucco, G.; Setragno, F. (2021). Multiclass semantic segmentation for digitisation of movable heritage using deep learning techniques. Virtual Archaeology Review. 12(25):85-98. https://doi.org/10.4995/var.2021.15329OJS85981225Adami, A., Balletti, C., Fassi, F., Fregonese, L., Guerra, F., Taffurelli, L., Vernier, P. (2015). The bust of Francesco II Gonzaga: From digital documentation to 3D printing. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, II-5/W3, 9-15. https://doi.org/10.5194/isprsannals-II-5-W3-9-2015Badrinarayanan, V., Kendall, A., Cipolla, R. (2017). Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481-2495. https://doi.org/10.1109/TPAMI.2016.2644615Balletti, C., Ballarin, M., & Guerra, F. (2017). 3D printing: state of the art and future perspectives. Journal of Cultural Heritage, 26,172-182. https://doi.org/10.1016/j.culher.2017.02.010Balletti, C., & Ballarin, M. (2019). An application of integrated 3D technologies for replicas in Cultural Heritage. International Journal of Geo-Information, 8(6), 285. https://doi.org/10.3390/ijgi8060285Barbieri, L., Bruno, F., & Muzzupappa, M. (2018). User-centered design of a virtual reality exhibit for archaeological museums. International Journal on Interactive Design and Manufacturing (IJIDeM), 12, 561-571. https://doi.org/10.1007/s12008-017-0414-zCaruana, R., Lawrence, S., & Giles, C. L. (2001). Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. Advances in Neural Information Processing Systems (pp. 402-408). https://doi.org/10.1109/IJCNN.2000.857823Cermelli, F., Mancini, M., Bulo, S. R., Ricci, E., & Caputo, B. (2020). Modeling the background for incremental learning in semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 9233-9242. https://doi.org/10.1109/CVPR42600.2020.00925Condorelli, F., Rinaudo, F., Salvadore, F., & Tagliaventi, S. (2020). A neural network approach to detecting lost heritage in historical video. International Journal of Geo-Information, 9(5), 297. https://doi.org/10.3390/ijgi9050297Chiabrando, F., Sammartano, G., Spanò, A., & Spreafico, A. (2019). Hybrid 3D models: When Geomatics innovations meet extensive built heritage complexes. International Journal of Geo-Information, 8(3), 124. https://doi.org/10.3390/ijgi8030124Dall'Asta, E., Bruno, N., Bigliardi, G., Zerbi, A., & Roncella, R.(2016). Photogrammetric techniques for promotion of archaeological Heritage: the Archaeological Museum of Parma (Italy). International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLI-B5, 243-250. https://doi.org/10.5194/isprs-archives-XLI-B5-243-2016Felicetti, A., Paolanti, M., Zingaretti, P., Pierdicca, R., & Malinverni, E. S. (2020). Mo.Se.: Mosaic image segmentation based on deep cascading learning. Virtual Archaeology Review, 12(24), 25-38. https://doi.org/10.4995/var.2021.14179Fiorucci, M., Khoroshiltseva, M., Pontil, M., Traviglia, A., Del Bue, A., & James, S. (2020). Machine Learning for Cultural Heritage: A Survey. Pattern Recognition Letters, 133, 102-108. https://doi.org/10.1016/j.patrec.2020.02.017Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., & Garcia-Rodriguez, J. (2017). A survey on deep learning techniques for image and video semantic segmentation. Applied Soft Computing, 70, 41-65. https://doi.org/10.1016/j.asoc.2018.05.018George, D., Xie, X., & Tam, G. K. (2018). 3D mesh segmentation via multi-branch 1D convolutional neural networks. Graphical Models, 96, 1-10. https://doi.org/10.1016/j.gmod.2018.01.001Giuffrida, D., Mollica Nardo, V., Giacobello, F., Adinolfi, O., Mastelloni, M. A., Toscano, G., & Ponterio, R. S. (2019). Combined 3D surveying and Raman Spectroscopy Techniques on artifacts preserved at Archaeological Musem of Lipari. Heritage, 2(3), 2017-2027. https://doi.org/10.3390/heritage2030121Grilli, E., Farella, E. M., Torresani, A., & Remondino, F. (2019). Geometric features analysis for the classification of Cultural Heritage point clouds. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W15, 541-548. https://doi.org/10.5194/isprs-archives-XLII-2-W15-541-2019Grilli, E., Özdemir, E., & Remondino, F. (2019). Application of machine and deep learning strategies for the classification of Heritage point clouds. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-4/W18, 447-454. https://doi.org/10.5194/isprs-archives-XLII-4-W18-447-2019Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.Gu, J., Wang, Z., Kuen, J., Ma., L., Shahroudy, A., Shuai, B., & Chen., T. (2018). Recent advances in convolutional neural networks. Pattern Recognition, 77, 354-377. https://doi.org/10.1016/j.patcog.2017.10.013Guidi, G., Malik, U. S., Frischer, B., Barandoni, C., & Paolucci, F. (2017). The Indiana University-Uffizi project: Metrological challenges and workflow for massive 3D digitization of sculptures. 23rd International Conference on Virtual System & Multimedia (VSMM), 1-8. https://doi.org/10.1109/VSMM.2017.8346268He, T., Shen, C., Tian, Z., Gong, D., Sun, C., & Yan, Y. (2019). Knowledge adaptation for efficient semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 578-587. https://doi.org/10.1109/CVPR.2019.00067Jégou, S., Drozdzal, M., Vazquez, D., Romero, A., & Bengio, Y. (2017). The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 11-19). https://doi.org/10.1109/CVPRW.2017.156Kersten, T. P., Tschirschwitz, F., & Deggim, S. (2017). Development of a virtual museum including a 4D presentation of building history in Virtual Reality. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W3, 361-367. https://doi.org/10.5194/isprs-archives-XLII-2-W3-361-2017Knyaz, A. V., Kniaz, V. V., Remondino, F., Zheltov, S. Y., & Gruen, A. (2020). 3D reconstruction of a complex grid structure combining UAS images and deep learning. Remote Sensing, 12(19), 3128. https://doi.org/10.3390/rs12193128Lin, P., Sun, P., Cheng, G., Xie, S., Li, X., & Shi, J. (2020). Graph-guided architecture search for real-time semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4203-4212. https://doi.org/10.1109/CVPR42600.2020.00426Llamas, J., Lerones, P. M., Medina, R., Zalama, E., & Gómez-García-Bermejo, J. (2017). Classification of architectural heritage images using deep learning techniques. Applied Science, 7(10), 992. https://doi.org/10.3390/app7100992Lo Turco, M., Piumatti, P., Rinaudo, F., Tamborrino, R., & González-Aguilera, D., (2018). B.A.C.K. TO T.H.E. F.U.T.U.RE. − BIM acquisition as cultural key to transfer heritage of ancient Egypt for many uses to many users replayed. In S. Bertocci (Ed.), Programmi Multidisciplinari Per L'internazionalizzazione Della Ricerca. Patrimonio Culturale, Architettura e Paesaggio (pp. 107-109). DIDA Press.Lo Turco, M., Piumatti, P., Rinaudo, F., Calvano, M., Spreafico, A., & Patrucco, G. (2018). The digitisation of museum collections for research, management and enhancement of tangible and intangible heritage. 3rd Digital Heritage International Congress (DigitalHERITAGE) held jointly with 24th International Conference on Virtual Systems & Multimedia (VSMM 2018), San Francisco, CA, USA. https://doi.org/10.1109/DigitalHeritage.2018.8810128Mafrici, N., & Giovannini, E. C. (2020). Digitalizing data: From the historical research to data modelling for a (digital) collection documentation. In M. Lo Turco, E. C. Giovannini, , & N. Mafrici (Eds.), Digital & Documentation. Digital Strategies for Cultural Heritage (Vol. 2, pp. 38-51). Pavia University Press. https://doi.org/10.5194/isprs-archives-XLII-2-W15-519-2019Malik, U. S., Guidi, G. (2018). Massive 3D digitization of sculptures: Methodological approaches for improving efficiency. IOP Conference Series: Material Science and Engineering, 364. https://doi.org/10.1088/1757-899X/364/1/012015Minto, S., & Remondino, F. (2014). Online access and sharing of reality-based 3D models. SCIRES-IT-SCIentific RESearch and Information Technology, 4(2), 17-28. http://doi.org/10.2423/i22394303v4n2p17Patrucco, G., Chiabrando, F., Dondi, P, & Malagodi, M. (2018). Image and range-based 3D acquisition and modeling of popular musical instruments. Proceedings from the Document Academy, 5(2), 9. https://doi.org/10.35492/docam/5/2/9Patrucco, G., Rinaudo, F., & Spreafico, A. (2019). A new handheld scanner for 3D survey of small artifacts: The Stonex F6. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W15, 895-901. https://doi.org/10.5194/isprs-archives-XLII-2-W15-895-2019Pierdicca, R., Paolanti, M., Matrone, F., Martini, M., Morbidoni, C., Malinverni, E. S., Frontoni, E., & Lingua, A. M. (2020). Point cloud semantic segmentation using a deep learning framework for Cultural Heritage. Remote Sensing, 12(6), 1005. https://doi.org/10.3390/rs12061005Salvador-García, E., Viñals, M. J., & García-Valldecabres, J. L. (2020). Potential of HBIM to improve the efficiency of visitor flow management in Heritage sites. Towards smart heritage management. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLIV-M-1-2020, 451-456. https://doi.org/10.5194/isprs-archives-XLIV-M-1-2020-451-2020Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1), 1-48. https://doi.org/10.1186/s40537-019-0197-0Stathopoulou, E. K., & Remondino, F. (2019). Semantic photogrammetry: Boosting image-based 3D reconstruction with semantic labeling. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W9, 685-690. https://doi.org/10.5194/isprs-archives-XLII-2-W9-685-2019UNESCO. (1979). Recommendation for the Protection of Movable Cultural Property, Records of the General Conference, 20th Session, I: Resolutions. Paris: UNESCO.Vargas, R., Mosavi, A., & Ruiz, R. (2018). Deep learning: A review. Advances in Intelligent Systems and Computing, 29(8), 232-244. https://doi.org/10.20944/PREPRINTS201810.0218.V1Yazan, E., & Talu, M. F. (2017). Comparison of the stochastic gradient descent based optimization techniques. 2017 International Artificial Intelligence and Data Processing Symposium (IDAP), 1-5. https://doi.org/10.1109/IDAP.2017.809029

Structure-from-Motion (SFM) Photogrammetry as a Non-Invasive Methodology to Digitalize Historical Documents: A Highly Flexible and Low-Cost Approach?

Historical documents represent a significant part of the world cultural heritage and need to be preserved from physical deformation due to ageing. The restoration of fragile documents requires economic resources that are often limited to only preserve the integrity of exceptional and highly valuable historical records. On the other hand, regeneration of ancient documents in digital form is a useful way to preserve them regardless of the material they are made of. In addition, the digitization of historical cartography allows creating a valuable dataset for a variety of GIS applications as well as spatial and landscape studies. Nonetheless, historical maps are usually deformed, and a contact-scanning process could damage them because this method requires planar positioning of the map. In this regard, photogrammetry has been used successfully as a non-invasive method to digitize historical documentation. The purpose of this research is to assess a low-cost and highly flexible strategy to digitize historical maps and documents through digital photogrammetry using low-cost commercial off-the-shelf sensors. This methodology allows training a wider audience of cultural heritage operators in digitizing historic records with a millimeter-level accuracy

TLS AND IMAGE-BASED ACQUISITION GEOMETRY FOR EVALUATING SURFACE CHARACTERIZATION

In the framework of cultural heritage documentation projects, it is very frequent the use of dense and detailed 3D models derived from range-based or image-based techniques. These reality-based models represent an effective and powerful solution to document the geometries, the surfaces and the characteristic of heritage assets. In fact, these technologies allow us to accurately describe the level of detail and the surface characterization of the materials, and also to provide precious support for the evaluation of the conservation status of the surveyed structures. In particular, the object studied in the presented research experience is the Morano sul Po arch (Morano sul Po, AL, Piedmont, Italy), a valuable example of an industrial archaeology asset. For the knowledge process of the arch, both LiDAR systems and photogrammetric strategies have been used, in order to properly document the geometry, the consistency of the material and the decays. The specific aim of the research presented in this paper is the evaluation of the advantages and the critical issues of using a telescopic pneumatic pole to raise the position of the scans from the ground and decrease the angle of incidence of the laser beam on the surveyed object. For this reason, the study also takes into consideration the use of mini UAVs and their flexibility to effectively acquire vertical surfaces even at elevated heights, comparing the density and the roughness of the derived model with the data obtainable by using TLS systems

Digital models of architectural models: from the acquisition to the dissemination

Antiquities and works of art preserved in museum collections represent an invaluable evidence of our history. A proper three-dimensional metric survey and digitisation of these assets (which are intrinsically fragile and for this reason need a continue and careful documentation) allow to increase significantly their resilience and they offer a valid contribution for the management of these objects belonging to movable heritage. This work takes place during a research experience carried out in the framework of B.A.C.K. TO T.H.E. F.U.T.U.RE. (BIM Acquisitions as Cultural Key TO Transfer Heritage of ancient Egypt For many Uses to many Users REplayed) during which a collection that consists of fourteen wooden models belonging to Museo Egizio of Torino has been digitised using both imagebased and range-based modeling techniques. In addition to geometric and radiometric data, provided by textured model, information of various nature related to the considered asset has been integrated. The main aim of the research, starting from the digital 3D models, is the creation of threedimensional databases (with alphanumeric and multimedia informations about historical, artistic and management aspects), useful for several purposes: 3D visualisation, communication, dissemination and data management. In this paper 3D metric acquisition strategies have been evaluated and the followed methodology as regards data enrichment have been illustrated

Strategie sostenibili a supporto del processo conoscitivo di insediamenti alpini abbandonati: un rilievo low-cost per il recupero della borgata Coletta (VB)

Le borgate abbandonate situate sul territorio alpino rappresentano un prezioso patrimonio da recuperare e valorizzare. Molti progettisti si dedicano a questa tematica e la Geomatica può offrire un valido contributo al processo conoscitivo di questi beni. Il presente contributo si pone l’obiettivo di descrivere un approccio sostenibile al rilievo per la documentazione metrica di una borgata alpina abbandonata con l’ausilio, oltre che di tradizionali tecniche topografiche, di un drone commerciale e di uno smartphone

Developing software beyond customer needs and plans: An exploratory study of its forms and individual-level drivers

Excessive software development is the tendency to develop new software above and beyond the requirements of the market and/or planned specifications. It is a widespread phenomenon involving both risks and flexibility advantages. As it represents a challenging dilemma for software developers, it is important to study its human origins. Drawing on the tripartite model of individual attitudes, this study investigates the influence of developers’s cognitive (intuitive and rational thinking styles), affective (emotional attachment) and behavioural (reliance on past experiences) traits on two forms of excess, beyond needs and beyond plans. Using survey data on 307 software developers, this study shows that different manifestations of excess are associated with distinct traits of software developers. Emotional attachment drives beyond needs excess. A positive (negative) association is found between relying on past experiences and beyond needs excess (beyond plans excess). An intuitive cognitive style fosters the inclusion of extra features in the new product scope, whereas a rational style might lead to developing one-size-fits-all software that targets the needs of a broad user base. These findings contribute to research on the development of digital new products and production technologies by offering a comprehensive yet fine-grained picture of excessive software development’s nature and drivers

SFM-based 3D reconstruction of heritage assets using UAV thermal images

In the last few years, notable progress has been made in the field of non-invasive diagnostic for the monitoring of heritage assets. In particular, multispectral imagery (more specifically thermal images will be addressed in this manuscript) allows investigations in the non-visible range of the electro-magnetic spectrum to be effectively carried out. Many researchers are currently exploring the possibilities related to the use of this kind of images in photogrammetric SfM-based processes to produce 2D and 3D value-added metric products, characterised by high level of detail and spatial resolution, including the information connected to the non-visible data. A data fusion-based strategy enables co-registering visible and thermal images in order to exploit the higher spatial resolution of the traditional true colour images. However, there are still many shortcomings to be addressed to properly and efficiently orient TIR (Thermal Infrared) images, connected (among other factors) to their low spatial resolution, or to the low contrast between adjacent materials characterised by similar emissivity. This paper proposes two different workflows to process thermal images using SfM algorithms, applied to three different case studies, each characterised by different characteristics and features (size, morphology, emissivity of the materials, etc.). The different pipelines are described and the obtained results are critically evaluated considering the metric accuracy, 3D geometric reconstruction and noise, completeness of the data and overall quality of the generated dense point cloud. Additionally, the effectiveness of the adopted strategies in connection with the peculiar features of the analysed case studies is also considered

Low-cost sensors for rapid mapping of cultural heritage: first tests using a COTS Steadicamera

n/

THERMAL AND OPTICAL DATA FUSION SUPPORTING BUILT HERITAGE ANALYSES

The recent developments of passive sensors techniques, that have been able to take advantage of the technological innovations related to sensors technical features, sensor calibration, the use of UAV systems (Unmanned Aerial Vehicle), the integration of image matching techniques and SfM (Structure from Motion) algorithms, enable to exploit both thermal and optical data in multi-disciplinary projects. This synergy boost the application of Infrared Thermography (IRT) to new application domains, since the capability to provide thematic information of the analysed objects benefits from the typical advantages of data georeferencing and metric accuracy, being able to compare results investigating different phenomena. This paper presents a research activity in terrestrial and aerial (UAV) applications, aimed at generating photogrammetric products with certified and controlled geometric and thematic accuracy even when the acquisitions of thermal data were not initially designed for the photogrammetric process. The basic principle investigated and pursued is the processing of a photogrammetric block of images, including thermal IR and optical imagery, using the same reference system, which allows the use of co-registration algorithms. Such approach enabled the generation of radiance maps, orthoimagery and 3D models embedding the thermal information of the investigated surfaces, also known as texture mapping; these geospatial dataset are particularly useful in the context of the built Heritage documentation, characterised by complex analyses challenges that a perfect fit for investigations based on interdisciplinary approaches